VURL verifyKeyContent True when there are no recorded equivkeys

Avoid verification error when addurl --verifiable is used with an url
claimed by a special remote other than the web.

diff --git a/Backend/VURL.hs b/Backend/VURL.hs
index f1dfe38a4a0bd6a89344e6471295553c87cfeb05..2552d08bfcb83f56d4ed9e24b1ff14acad6ba649 100644 (file)
--- a/Backend/VURL.hs
+++ b/Backend/VURL.hs
@@ -31,8 +31,12 @@ backendVURL = Backend
                        -- Normally there will always be an key
                        -- recorded when a VURL's content is available,
                        -- because downloading the content from the web in
-                       -- the first place records one.
-                       [] -> return False
+                       -- the first place records one. But, when the
+                       -- content is downloaded from some other special
+                       -- remote that claims an url, that might not be the
+                       -- case. So, default to True when no key is
+                       -- recorded.
+                       [] -> return True
                        eks -> do
                                let check ek = getbackend ek >>= \case
                                        Nothing -> pure False

diff --git a/CHANGELOG b/CHANGELOG
index e6a89262c3422352e2e3bd997cfb0e1368f98262..cdef9716d2471e32a01f4a5c5d000a87b33ec857 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -3,6 +3,8 @@ git-annex (10.20250103) UNRELEASED; urgency=medium
   * Improve handing of ssh connection problems during 
     remote annex.uuid discovery.
   * log: Support --key, as well as --branch and --unused.
+  * Avoid verification error when addurl --verifiable is used
+    with an url claimed by a special remote other than the web.
 
  -- Joey Hess <id@joeyh.name>  Fri, 03 Jan 2025 14:30:38 -0400
 

diff --git a/doc/bugs/VURL_verification_failure_on_first_download.mdwn b/doc/bugs/VURL_verification_failure_on_first_download.mdwn
index 7cadb61389c3973704627c8e928828466b481a8e..5d37eae73d313048965ec6eae71adb22bd2e1056 100644 (file)
--- a/doc/bugs/VURL_verification_failure_on_first_download.mdwn
+++ b/doc/bugs/VURL_verification_failure_on_first_download.mdwn
@@ -1,3 +1,5 @@
+[[!meta title="VURL equivilant keys not populated when addurl is used with special remotes other than web"]]
+
 ### Please describe the problem.
 
 With an external special remote that handles a custom URL scheme, I receive a "Verification of content failed" on the first `git annex get` of a file (i.e. when git-annex cannot know a checksum for the file, yet).

diff --git a/doc/bugs/VURL_verification_failure_on_first_download/comment_1_fe5525d4c03fdafc28dc23ed5ee3fab8._comment b/doc/bugs/VURL_verification_failure_on_first_download/comment_1_fe5525d4c03fdafc28dc23ed5ee3fab8._comment
new file mode 100644 (file)
index 0000000..f1e146d
--- /dev/null
+++ b/doc/bugs/VURL_verification_failure_on_first_download/comment_1_fe5525d4c03fdafc28dc23ed5ee3fab8._comment
@@ -0,0 +1,49 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2025-01-03T19:52:28Z"
+ content="""
+Looking at [[!commit 55bf01b78888b410a8ac07e834ed7104ffa1f4d0]] it talks
+about something like this:
+
+       When downloading a VURL from the web, make sure that the equivilant key
+       log is populated.
+
+       ...
+
+       Downloads of VURL keys from other special remotes that claim urls,
+       like bittorrent for example, does not popilate the equivilant key log.
+       So for now, no checksum verification will be done for those.
+
+I think at the time, I just punted on the question of how to register the
+equivilant key when downloading a VURL from other special remotes.
+
+Thing is, git-annex just calls retrieveKeyFile at some point. At that
+point, it doesn't know if it's retrieving the key from the web or some
+other special remote that claims urls.
+
+It would not do for a `git-annex get` that happens to get the VURL key from
+eg, a directory special remote to say hey, we've not populated the
+equivilant key log for this VURL yet, let's trust what we downloaded from
+the remote is good and populate it now.
+
+So that's why the code went into the web special remote. It seems to me that to
+move it out of the web special remote, there would need to be a way for
+git-annex to check if a given special remote claims the url associated with a
+VURL. Eg, look up the key's url(s) and check claimUrl.
+
+Hmm, if it did that, there would still be a problem that retrieveKeyFile can
+try to verify the content it downloads, using verifyKeyContent. But when no
+equivilant key is yet registered for the VURL, retrieveKeyFile would find that
+anything it downloaded failed to verify. So it would fail. Chicken and egg problem.
+
+But well, it would be easy enough to make this not be treated as a failure.
+In Backend.VURL, verifyKeyContent returns False when there are no equivilant
+keys. It could just return true.
+
+That seems like a necessary first step. Just making that change will solve your
+problem to the extent that it's ok for no verification of the content of VURLs
+to be done when they originate with your special remote.
+
+So I've done that. Whether it makes sense to do the rest I am not yet sure..
+"""]]

diff --git a/doc/git-annex-addurl.mdwn b/doc/git-annex-addurl.mdwn
index 769bed69a756bbfcefe8ea2100cb8e0219a84c37..c2247e1de960c2c7f9a00dd5ede1a7338428bac6 100644 (file)
--- a/doc/git-annex-addurl.mdwn
+++ b/doc/git-annex-addurl.mdwn
@@ -50,12 +50,14 @@ be used to get better filenames.
   verified with a checksum when it is transferred between git-annex
   repositories, as well as by things like `git-annex fsck`.
 
-  When used with --relaxed, content from the web will always be accepted,
-  even if it has changed, and the checksum recorded for later verification.
+  When used with --relaxed, content from the web special remote will
+  always be accepted, even if it has changed, and the checksum recorded
+  for later verification.
 
   When used with --fast, the checksum is recorded the first time the
-  content is downloaded from the web. Once a checksum has been recorded,
-  subsequent downloads from the web must have the same checksum.
+  content is downloaded from the web special remote. Once a checksum has
+  been recorded, subsequent downloads from the web special remote 
+  must have the same checksum.
 
   When addurl was used without this option before, the file it added 
   can be converted to be verifiable by migrating it to the VURL backend.

diff --git a/doc/todo/migration_to_VURL_by_default/comment_2_83b03035f1cae24180038f412e1886f2._comment b/doc/todo/migration_to_VURL_by_default/comment_2_83b03035f1cae24180038f412e1886f2._comment
new file mode 100644 (file)
index 0000000..277be82
--- /dev/null
+++ b/doc/todo/migration_to_VURL_by_default/comment_2_83b03035f1cae24180038f412e1886f2._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2025-01-03T21:17:31Z"
+ content="""
+I think that is no longer a blocker for this becoming default, it no longer
+causes a verification failure, instead the content originating with such a
+special remote for a VURL is effectively treated the same as for an URL,
+and just doesn't get verified in transit later on.
+"""]]